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ABSTRACT 


Tiiis paper reports on a usually unrecognized 
pot-ntial benefit of measurement instriments designed for research in 
instr iction: the effectiveness of these instruments When used as 
instructional procedures. Two studies were involved in the 
development of this -concept. In the first study, an extensive 
instrument, was designed- to measure changes in six variables : 
considered indicative of improvement in skills and qualities in 
narrative writing. The instrument was used to> measure t^.e 
effectiveness- of a sel^-instructional program which employed .a number 
of instructional strategies. The second study focused on one of' the 
instructional strategies: task-related feedback, or ^information bn 
performance that is. dire'ctly related'to specific behaviors identified 
by.means of a task analysis of.the performance of narrative writing. 
Th? measurement instrument was simplified and transformed, into a 
feedback system used by students and teachers. Finiings supported the 
idea’ that task-related feedback could result in H des irable changes. 

The implication of tbe study was that the u^e of the. f / 

feedback/measurement System, by itself, could be an effective 
instructional procedure. (Author/LJR) . 
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The Instructional Use of Measurement' Instruments 
• . Designed for Research , 


ABSTRACT 


Researchers in education expend much time and effort, in 

C . 

deciding upon variables to measure, developing measurement 

* ' kV . ' 5? 

instruments^ and tes t i nrf. the i r validity and reliability. A 

'* * 

potential henefip of those•instrumentsis in their use as # 

i 

instructional procedures. Two investigatorsworking on 
instructional* problems invoVved in the improvement of narrative 

* b . * , ‘ „ 

writ-ing ^-secondary and elementary students, discovered the' 
benefit to the^r oWh .area. One investigator developed an 

* ’* * . » y * . 

elaborate, precise instrument for measuring six positive changes 

i-ndicativ^ bf improvement in narrative wr?ting. The second 

investigator adapted the instrument as 'feedback system. The, 

' * • * 

% f # * 
findings of both stydies lend support to the idea that the 

• V * V 1 , 

providing of specific, positive, task-related feedback (or 
measurement) is, -in itself, an effective instructional, procedure 



The I nst-ruet iona I Use of Measurement 
Instruments Designed for Research- 


) 


This paper reports on a widely unrecognized., potential benefit 

* . i 

e> • * ^ 

of measurement instrumehts designed for research in instruction: 

0 v. . • ^ 

the effectiveness of these instruments when used as * instructional 

. o 

procedures. .This benefit was an accidental discovery made by two 

»- 

investigators working on related problems (involved in the development 


and-testing of instructiona] strategies for improving narrative w'riting 


by elementary and secondary.school students). In one study, a rather 


extensive instrument was designed to measure changes in six variables 

ci % 

< + • - 

cons idened. ind i cat ive of improvement in'skills' and qualities in 

• "* 0 

narrative writing.' The instrument was used to measure the effgctive- 

1 . . ( 

ness of a self-instructiohal program which employed a number of y * . 

a X • 

* * • % ■ 9 
instructional strategies. The seco.nd study focused on ond of the 

instructional strategies, task-related feedback; i.e., Information 

• < 

on performance that is dirpctly related to-specific behaviors 

V 

identified hy means, of a task analysis of the performance ofjiarrative 

, ’ k 

0 

writ.ing. For that study, the measurement instrument was simplified 

(7 

/ 

and transformed into*a feedback system, used by students and teachers. 

0 

. * . • b 

Findings lent support to the idea that task-related feedback coujd 

o - * ' 

» ' * , 

resul t* in des'i rabl,e changes. The implication of the Study was that 

* . . ' ' ' 

the use of the feedback/measurement system, by itseff, could be an 

* *# 

. V * * 

effective instructional procedure. ' c 

• • - * * • 

• 0 - 

A narrative-summary of some°of the elements of the two studies 

-i 

wil*h be "provided to explain the points-made above. Though the two 
studies were very limited (in terms, (a), of numbers of subjects and 


/■ 










(h) of scope, i.e., only one type of writing and only a few dualities 
of writing), the implications seem important ^to research in instruction 
in any area. The implications also suggest a way that teachers can 

t 

' * £ 

take advantage of the work of researchers in improving^ instruction:,/ 

• . • ^ 

by using°the measurement instruments as a-basis for (a) identifying 

«... 

meaningful and measurable objectives and (b) hy providing feedback - 

o • ^ w « 

directly related.to those objectives. ' 


The Heas.urement Problem in’ W r,i t ing . . 

IJntM recently, and in only a few instances (Buxton, 1998; Harris, 10f> 
. 1970^ Lamherg, 1^74; Wolter, 19?5) , there has been no attempt- to.design 

specific', objective measures of positive behaviors in writ’jnn., A 

' ‘ :.•/*. ... " ; 
reading of the major review and critique of research in writing 

(Braddock, Ll-oyd-Jones, 6 Schoer, 19^3)., as well as more recent 

.• . * • 

reviews such as Sherwin (19^9), deads one to the following conclusions 
about most of the research conducted: (1) In the past, much effort 


was devoted to the elimination of negative behavioVs; i.e., errors in 

spelling, punctuation, ^sentence structure, and usage. Specific 

■ * % ^ * 
•measures were’used, but (a) most teachers consider such errors of little- 

. • .• 1 ^ 

importance compared to ot(jer\ski 1 Is and qualities in writing, and* (h). 

studies found that instructional procedures aimed at eliminating errors ' 

• * 

were.-la.cking in effectiveness. ■ , 

c 

• * ^ 

(2) Studies that focused on improvement in. or acquisition of 

* J 

'desirable, qualities, typically,- used instruments that involved subjective. 


norm-referenced evaluations of hard-to-define, 


and often undefined, . » 



qualities, e.g., significance, sincerity, critical thinking. (3) Few 

studies were specifically concerned with measurement and few showed, 

'♦ 

or reported any, consideration of measurement; most*that did were 
concerned with determining (and/or increasing) the re 1iabi]ity-of 
. subjective measures., (k) Few studies were concerned with (and 
therefore did not measure) b asi.c ski.lls and qualities, e.;g., fluency 

in writing words, fluency in writing complete sentences, unit°y, 

• > 

completeness. These qual ities are more readily, defined and, 

k 4 

% 

perhaps, prerequisitic to the sophisticated "adult standards" more 

commonly used, to evaluate writing, like those noted in (?) above. 

* ' . • # 

It should he noted that the trends in oast research reflect the 

trends inclassroom practice. The elimination of*errors has typically 

« 

received the greatest amount O o f attention; feedback when specific has 

been usually negative’; i.e., rioting errors; and most feedback has been 

V'*. 

, negativd; positive feedback and*positive evaluations have been, general ly 

subjective (Lamberq, 197**; Wolter', 1°75). 11 

♦ 

Students' writing, typically, has been evaluated by means of 

norm-referenced, subjective measures (how well the students' writing 

• « 

o • ^ «. 

compares with*their classmates’’in terms of often undefined qualities) 

' » G - 

rather than in terms of changes in specific behaviors measured by. 
criterion-referenced instruments. • > 


An Attempt to Identify and Measure Specific, Positive Changes. 

An attempt was <nade to design and test a measuremerit instrument, 
that would provide objective measures of positive changes in specific 

t • * 

behaviors considered indicative of improvement in .the short, auto- 
.biographical narrative (LambeVg, 197*0. ' The.instrument was used to 




test the effectiveness, and to aid in the development of, a setf* 

instructional, program that employed a nunber of instructional strategies: 

(l) task-analysis (the attempt to identify b'asic elements in narrative 

writing and specific objectives for improvement), (?.) discrimination 
. • \ 

programming (Smith, a strategy for designing instructional 

materials directly related to the task analysis, (3) the teaching'of a 

» 

process-approach, by which the student is taught a sequence of steps .. 

* • * , * 

- for (a) selecting ap appropriate subject, (b) developing it j-n a rough- 

O 

draft, and (c) revising and expanding ‘i t, and »(*») the use of a feedback 
system, by which the jstudent could provide himself with positive 

r"' 

information on his improvement.' 

* . • • . 

The first step in developing ,the measurement instrument was a 

■ > . , ’ *■ 

rough task-analysis of-narrative writing. -The task-analy$is defined 
, a narrative as a discourse that answered basic questions: What was. 

, it about? What happened?' Why did it happen? When did it happen? 

Where did it happen? Who did it happen to? Who else was involved? 

The task analysis further identified the -basic writing responses: 

' a student would write to provide appropriate answers to those basic 
questions.- 

^ The difficult problem was to specify what "appropriate ’answers" 

were or, put another way, what*desirable changes in answering the 

questions would be. Basic goals were identified for improvement in 

narrative writing, based on the work of Moffe.tt (l^fB). Two goals 

' were identified: control of structure,, accomplished simultaneously 

* 

with expansion. Simply put, the student would learn to expand his 

. * ' 

narratives, by writing more words for the sake of. completeness, 
development, and Interest, without sacrificing control of unity and-orde 



I 


The next step was to define the qualities of control Of structure 


and expansion.. For the formef, three factors were identified: (1) 

i 

unity (the narrative. would be ahout one incident that oedurred within 
twentv-four hours): {?.) consistency o^f point ‘Qf view (since the 
narrative was autobiographical it would h ffi "toln" in the f i rst'-person); 


and (3) chronological or^er (events would tye narrated i-n the order in 


which they happened),. Expansion was "hroken down" into two factors: 


(1) "Completeness 1 '' (there would he an increase in the number of questions 


answered to some extent)"*, and (2) "heve1opment" (there would he an increase 


*in number of words that answered each of the seven questions). In 


addition, expansion was measured in an easier <yay, by an increase .in 


total number of words*. (It was found th^t the increase in number of , 


words > had a high correlation with increases in "completeness" and 


"development".) 


The last step was to’wrjte/a detailed set of instructions for 


judges'who measured the narratives, one which completely and precisely 


defined the factors note^ above. Judges were trained in us.ng the 


instrument with sample narratives; The instrument served as the 


clearest statement of the objectives of'the i nst ruct ion, _ 


defined in measurable terms the desired positive changes. Though*time¬ 


consuming and complex, the instrument was found to Have a hmqh deqree 


of rel iab i 1 i ty 


The Use of the Instrument as a Feedback System. 


The self-instructional program was found to have statistically 
significant changes when pre and post-test narratives were compared. 


Variables, however, were, not controlled. The second investigator 


(Wolter, 1975) ’decided to continue the work on instruction in narrative 

• , . .8 - - - ■ . . 










writing hut to limit the study to, and provide thorough control for, v 

one variable: task-re'ated feedback. Four groups of students wrote 

_ _ , 

narra<(tIv.es,^folJovti-ng the same instructions (which were; based upon, the 

task-analys is). 

# 

The effects of four conditions of feedback were studied. One 
group received positive, quantitative feedback from the teacher on a 
form, i.e., number of words written, number of questions answered. 

A second group received corrective feedback by means of written comments 
from the teacher 1 , based on the form, i.e., specific recommendations for 
'improving the narrative by adding* particular details, answering particular 
questions or y/riting more words. A third group provided themselves 

' ft • ' , * 

with positive, quantitative feedback by recording information about 
their narratives on the feedback form. The fourth group received no 

0 f 

♦ 

feedback of any kind. 

To insure that the content of the feedback wSs the same for the 

• • 
three groups, was task-related, and was provided in concrete, objective 

0 

terms, the investigator hit upon tjie idea of^using the measurement 
system, in a’simplified form, and of training both the students and 
teachers in the use of the instrument. .Thus, the instrument, originally 
designed solely for research, had become an instructional 'procedure. 

• ^ 9 ^ w 

The feedback form is presented in'Tigure 1, 

The investigator found that any manner of providing task-related 

-V v • • . " 

feedback was more effective -than no feedback at aT1: “the cjrsups 

receiving feedback demonstrated stat i stical ly si.gni f ican-t changes. 

Task-related feedback was not’compared with nonrtask-related feedbadk 

‘ « • \ 

(i.e.., information dg aspects of performance not directly re'T^f.ed to 



- 7 - 


r 


Aj 


the basic elements of the task; in this case, spell inn, dunctuatioh, 
significance of the subject). However, the implications of the study 
are that infohmation directly related to the task, in itself.,'has a * 
positive instructional effect. It should be noted that the students 
received no oth»r i ns t rue t i on; .g. , lectures on narrative writinq, 

r , 

discussions of wel 1 -wr i tten *narrat i-ves,, reading of wellgwritten 
narratives. It should also he noted that the group who provided, 
themselves .w?th positive feedback nay have achieved more, in some 

» 

respects, than the| two groups who (a) received positive quantitative 
feedback from teachers or .(b) corrective feedback from teachers. 

Ml students in the first group shaved positive changes, whereas in 

* N t o 

the Mother groups, some did not, artd the students in the first group’ 
were observed to spend more time working on and reading over their 
writing than the others., . 

' 1 

• i 

Implicat ions. - 

Researchers' expend much time and energy in deciding upon variables 

« ' * N 

to measure, developing measurement instruments, and testing their 

• • • « • * * 
validity and- rel iahi 1 i ty. The value of thi s .labor, s ince it is done. . 

% 

♦ as a means to an end, may be often overlooked, especial 1y by classroom 

. * r ' 

3 teachers*. Meanwhile* teaclfers expend much time and work trying out- 

• \ s >» ^ 

i' different, often elaborate procedures and ma terials to i mpro ve instruction 

» ft* * * . ” 

and expend, at leaftt in teaching writing, much time and energy in 

writing responses to students' performances. In spite of that { labor, 

•» * 

researchers and teachers of writing have been, discouraged about the 

% possibi 1 i t|ie,s of improving instruction in Writing. 'One answer, one > 

that seems in hindsight te'rribly simple # and common-sens leal, may be: 

• f 

! .10 




tb.glve students the’opportuni ty to measure their Own workj^by providing 

i * 

themselves with feedback; that is directly related to the tasks they are 

a 

working on. The researcher car^ be of great assistance in helping the 

0 

teacher^ identify the kind of feedback most beneficial to the -students. . 
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tiie.Cr. your story now, 


^' bow many words? 


( I Rox A 


See how well yourstory answers these questions. Put the points for each 
question in the box. ® . . 


1 


Vlliat vvas i-t about? ' •• • • 

aT ^icj* you tell about ,one experience? Then give 2 pts. 

b) Rid; it liappen in one day or less? Then give 1 pt. 

c) Didj it happen to you, OR to you and someone else? 

Then give 2 pts. " 

• *. i t 

Add the boxes 


□ 

□ 

□ 


I j Pj Box 1 


T> 

J • 


'What hap pened? 

Rive l.pt for each thing that happened. 
Stop counting if you get 10. 

Why did it happen? 


I 1 Box 2 


Reasons or causes'that tel \ why are often signalled by 
words like since, be cause , so. Give 1 pt. each time you 
use those words. (Stop at 5 pts.) 


□ r °* 7 


h. 


Who d i.d i t happen to? c * 

each piece of TrTformation about the main person like name , 
ane, apncarance, relationship with others. Gets 1 pt, 
TSTopTr you get r , ptsT) 


It Box h 


Who, else was irivol ved? v 

Each pidee of information thatctells the person's relation¬ 
ship' tg the main person, or tells of an action done b-y 
that person. Gets ^f'ptv-istop at 2 pts..) 


□ Ro * 


When fl fd_ Jj: happen? 

a)~Time words like morning. 


night, dayT^OACpning , in o'clock, □ 


5. minutes , get 1 pt. feachu * 

b) ATso, t’ime signal words like when, after , 1 ater , before , 
soon, get 1 pt. each 


Add the boxes (stop at 5 pts.) 


1 1 |_j* Box A 

“ / " dr- 1 


7. 


'..'here did it happen? 

G*Tv*e 1 ~ointT for each detail about the place where the 
story happened. Stop at 5 points. 


□ 


^ox f 


„:vqw AGO pG 1 NTS in b^(es 1 to 7 • * TOTAL 

* * * 

r,ow put your score from Box A. on the Graph A. 

o 

Then put your score -from Box B. on Graph B. 


n Box n. 


‘ i 
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Feedback form used 
task-related 


by teachers 
feedback on 


and students to provide 
narrative writing. 
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